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DETAILED ACTION 



Drawings 



1 . Figure 1 should be designated by a legend such as -Prior Art-- because only 
that which is old is illustrated. See MPEP § 608.02(g). A proposed drawing correction 
or corrected drawings are required in reply to the Office action to avoid abandonment of 
the application. The objection to the drawings will not be held in abeyance. 

2. The drawings are objected to as failing to comply with 37 CFR 1 .84(p)(5) 
because they do not include the following reference sign(s) mentioned in the 
description: 

Fig. 1 1 C, see page 9 line 25; and 

Figure 12 120 is not shown as described at page 10 line 5 and page 39 line 20. 

A proposed drawing correction or corrected drawings are required in reply to the 
Office action to avoid abandonment of the application. The objection to the drawings 
will not be held in abeyance. 

3. The drawings are objected to because: 

In figure 27 block 2706 line 3 "he" should be "the"; and 

Figure 23 block 2308 lines 4-5 "th irtual" should be "the virtual" and at block 2318 
"sight o the" should be "sight of the". A proposed drawing correction or corrected 
drawings are required in reply to the Office action to avoid abandonment of the 
application. The objection to the drawings will not be held in abeyance. 
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Specification 



4. The disclosure is objected to because of the following informalities: 
page 54 line 8 "he" should be "the". 

Appropriate correction is required. 

Claim Rejections - 35 USC § 102 

5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed 
publication in this or a foreign country, before the invention thereof by the applicant for a patent. 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 



6. Claims 1-6, 10-12, 23-27, 35, 40, and 41 are rejected under 35 U.S.C. 102(b) as 
being anticipated by Navab, U.S. Patent No. 6,028,912. 

The Navab patent teaches using a fluoroscope to analyze a patient and the 
orientation of a biopsy needle and the system displays on a display an image of a 
portion of the patient's body having the target turner, a target T identified by the 
surgeon, and a virtual extension of the needle, see column 6 lines 21-30. 



States. 



A detailed analysis of the claims follows. 
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Claim 1: 

Navab teaches a method for augmented reality guided instrument positioning 
(column 1 lines 5-11 and column 4 lines 33-64) comprising the Steps of: 
establishing a viewpoint (the surgeon establishes a viewpoint that will 
lead to the target point t), from which a line of sight to a point on a target 
(target point T) defines a path (from the surgeons viewpoint to target 
point T) for an instrument (needle) to follow during a positioning (as the surgeon 
adjusts the angle of the needle the displayed virtual line moves 
so the surgeon can move the needle until the virtual line 
intersects the target point T, column 4 lines 33-64 and column 6 
lines 21-30, also in the automatic mode the system determines 
the alignment of the needle) of the instrument (needle) to the point on the 
target (target point t); and aligning the instrument along the line of sight to the 
point on the target ( the surgeon adjusts the angle of the needle until 
the needle is aligned such that its line of sight is to the 
target point T in the manual mode or if in the automatic mode 
the system aligns the needle, see column 11 line 24+). 

Claim 2: 

Navab teaches the method according to claim 1 , further comprising the step of 
moving the instrument along the path towards the point on the target, subsequent to 
said aligning step (after the surgeon or the system has aligned the 



Application/Control Number: 09/81 8,427 Page 5 

Art Unit: 2672 

needle the system moves the needle towards the target T). 
Claim 3: 

Navab teaches the method according to claim 1, further comprising the step of 
rendering the target as a graphics object (the target is illustrated on the 
display with computer generated graphical objects such as the 
shown ellipse around a circular point). 

Claim 4: 

Navab teaches the method according to claim 3, wherein the graphics object 
comprises a marker (the shown ellipse around a circular point is a 
marker) that marks the point on the target (the target is a tumor and the 
point T is a point on the tumor). 

Claim 5: 

Navab teaches the method according to claim 4, wherein the marker has a circular 
shape, and is centered on the point on the target (the point t is shown as a 
circular dot on the target point). 
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Claim 6: 

Navab teaches the method according to claim 5, wherein the circular shape is a 

ring (the ellipse is a circular ring because it is circular, see the following dictionary 

definition of circular found at dictionary.com). 

circu-lar 
adj. 

Of or relating to a circle. 

Shaped like or nearly like a circle; round. 

Moving in or forming a circle. 
Circuitous; roundabout: took a circular route to the office. 
Using a premise to prove a conclusion that in turn is used to prove the premise: a 
circular argument 

Defining one word in terms of another that is itself defined in terms of the first word. 
Addressed or distributed to a large number of persons. 

n. 

A printed advertisement, directive, or notice intended for mass distribution. 

The American Heritage® Dictionary of the English Language, Fourth Edition. Copyright © 2000 by Houghton Mifflin 
Company. Published by the Houghton Mifflin Company. All rights reserved. 
© 1996-2002 yourDictionary.com, Inc. All Rights Reserved. 



Claim 10: 

Navab teaches the method according to claim 1 , wherein said aligning step comprises 
the steps of: 

positioning a tip of the instrument on the path (point f ), at an entry point on a surface 
of a physical object corresponding to the target (at column 6 lines 25-28 Navab 
describes placing point f on or near the patient y s body)] and 
rotating the instrument around the tip on the entry point (at column 6 lines 25-28 
Navab describes the surgeon changing the direction of the needle) 
until the instrument is aligned with the line of sight (between point f and point 
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T) toward the point on the target (at column 6 lines 25-28 Navab describes 
the surgeon changing the direction of the needle until the the 
virtual extension of the needle passes through the image t of 
the patient) . 

Claim 11: 

Navab teaches the method according to claim 10, further comprising the step of 
marking {placing the needle at point f on the patient 's body marks 
the entry point of the needle) the entry point on the physical object. 

Claim 12: 

Navab teaches the method according to claim 11, wherein said entry point is marked by 
a graphical object (marking the point with a graphical object because 
the line between point f and point T is a virtual line and at 
least its intersection with the outer surface of the patient *s 
body is a graphical object since the intersection of the virtual 
line and patient *s body is visually different than the reset of 
the displayed outer surface of the patient y s body). 

Claim 23: 

Navab teaches a method for virtual reality guided instrument positioning 
(semiautomatic and automatic), comprising the steps of: defining a point on an 
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actual target {point T is defined by the surgeon) and an actual path to reach 
the point on the actual target (the system determines the path from point 
f to point t); tracking a pose of an actual instrument with respect to a pose of the 
actual target (the orientation of the needle relative to the target T 
is determined by the system); rendering a graphical representation of the actual 
instrument (an image representing the needle is shown on the display 
under the outer surface of the patient) and the actual target point (a 
virtual target point f is displayed on the display) to obtain a virtual 
instrument (the computer generated image of the needle is a virtual 
ins trument) and a virtual target point (the computer generated image of the 
target is a virtual target point), respectively, the graphical representation 
being rendered with respect to a virtual viewpoint (one end of the needle) from 
which a virtual line of sight to the virtual target point coincides with a virtual path for the 
virtual instrument to follow during a positioning of the actual instrument to the point on 
the actual target (as the surgeon or system changes the direction of 
the needle the displayed virtual line indicating the path the 
needle would follow changes direction accordingly), the virtual path 
corresponding to the actual path (the displayed virtual line corresponds 
to the actual path the needle will take if it is pressed into 
the patient 's body)] aligning the virtual instrument along the virtual line of sight to 
the virtual target point to align the actual instrument along the actual path (the 
surgeon or the system when in the automatic mode aligns the 
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needle along the virtual line of sight to target point T). 
Claim 24: 

Navab teaches the method according to claim 23, further comprising the step of moving 
the actual instrument along the actual path towards the point on the actual target, 
subsequent to said aligning step (after the surgeon in the semiautomatic 
step or the system in the automatic step aligns the needle to 
the path then the needle is moved along the path to the target 
point T). 

Claim 25: 

Navab teaches the method of claim 23, wherein the virtual target point has a circular 
shape (see the discussion given for claim 5). 

Claim 26: 

Navab teaches the method of claim 25, wherein the circular shape is a ring (see the 
discussion given for claim 6). 

Claim 27: 

Navab teaches the method of claim 23, wherein the virtual instrument comprises a 3D 
structure for line of sight alignment (the system views the needle with 3D 
coordinates thus the system views the needle as having 3D 
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structure for alignment of the needle in the 3D coordinates of 
the patient's body. The claim is not specific about the 3D 
structure). 

Claim 35: 

Navab teaches the method of claim 23, further comprising the step of rendering 
graphical information about a distance between the actual instrument and the point on 
the actual target (the length of the virtual line is equal to the 
distance between the needle and the target point T, this length 
is a graphical representation of the distance between the needle 
and the target point T). 

Claim 40: 

Navab teaches the method of claim 23, wherein the graphical representation from the 
virtual viewpoint is combined with another graphical representation from another virtual 
viewpoint looking at the virtual path from a side thereof (see figure n, column 7 
line 61 to column 8 line 1 and column 9 lines 20-37). 

Claim 41: 

Navab teaches the method of claim 23, wherein said graphical representation from the 
virtual viewpoint is combined with an augmented reality view (the needle 
represents the real needle about to be inserted into the patient 
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and the virtual line and virtual target T are added graphical 
representations to augment the real world). 



7. Claims 1-7, 10-13, 16-27, 35, and 41 are rejected under 35 U.S.C. 102(a) as 
being anticipated by Benedicte Bascle et al. f Needle Placement under X-ray 
Fluoroscopy using Perspective Invariants, IEEE Workshop on Mathematical methods in 
Biomedical Image Analysis Proceedings, June 11-12, 2000, pages 46-53. 

The Bascle article teaches orienting a needle and displaying an augmented 
reality view of a patient and needle by displaying a dotted line to indicate the path the 
needle will take with the current orientation of the needle, see figures 7a and 7b, and 
displays a circle and a cross hair to indicate the target in the patient, see figure 1 . 

A detailed analysis of the claims follows. 

Claim 1 

Bascle teaches a method for augmented reality guided instrument positioning (see 
figure 1), comprising the steps of: establishing a viewpoint (the point on the body 
where the surgeon places the robot y s needle is point F), from which a 
line of sight to a point on a target (the surgeon marks the target in the 
image displayed at the workstation, see figure l) defines a path for an 
instrument to follow during a positioning of the instrument to the point on the target (the 
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straight line path from point F to the target is the path the 
needle will follow): and aligning the instrument along the line of sight to the point 
on the target (the surgeon moves the needle, passive robot arm, or 
the system moves the needle with an active robot arm to align 
the needle along the line of sight between point F and the 
target). 

Claim 2: 

Bascle teaches the method according to claim 1 , further comprising the step of moving 
the instrument along the path towards the point on the target, subsequent to said 
aligning step (after the needle is aligned to the path that will take 
it to the target, the needle is moved towards the target as 
shown in figure 7). 

Claim 3: 

Bascle teaches the method according to claim 1 , further comprising the step of 
rendering the target as a graphics object (figure l shows the target as a 
circle and cross). 
Claim 4: 

Bascle teaches the method according to claim 3, wherein the graphics object comprises 
a marker that marks the point on the target (see figure l). 
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Claim 5: 

Bascle teaches the method according to claim 4, wherein the marker has a circular 
shape, and is centered on the point on the target (see figure i). 

Claim 6: 

Bascle teaches the method according to claim 5, wherein the circular shape is a ring 

(the circle at the target in figure 1 is a ring). 

Claim 7: 

Bascle teaches the method according to claim 1 , further comprising the step of marking 
the path with a plurality of graphical markers (figure shows the path marked by 
a dashed line) 

Claim 10: 

Bascle teaches the method according to claim 1 , wherein said aligning step comprises 
the steps of: positioning a tip of the instrument on the path (the surgeon places 
the tip of the needle at point F which is on the pa th), at an entry 
point on a surface of a physical object {patient 's body) corresponding to the target 
(the surgeon selects the target at the workstation, see figure l); 
and rotating the instrument around the tip on the entry point until the instrument is 
aligned with the line of sight toward the point on the target (the surgeon or the 
system rotates the needle until its path coincides with the path 
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from point F to the target). 
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Claim 11: 

Bascle teaches the method according to claim 10, further comprising the step of 
marking the entry point on the physical object (the needle is placed at entry 
point of the patient *s body which marks this location, see 
figures 1 and 7). 

Claim 12: 

Bascle teaches the method according to claim 1 1 , wherein said entry point is marked by 
a graphical object (the displayed needle guide shown in figure 7 is a 
graphical object at the entry point because it is an object 
displayed at the entry point of the needle). 

Claims 23-27, 35, and 41: 

Since Bascle is similar to Navab a separate discussion of how these claims are rejected 
by Bascle is not necessary. Please refer to the discussion of claims 23-27, 35, and 41 
given in the rejection based upon the Navab reference- 
Claims 13 and 16-22 do not claim the marker is a graphical marker on a virtual 
instrument corresponding to the actual instrument. Thus these claims are claiming 
markers on the real instrument or on a virtual instrument. 
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Claim 13: 

Bascle teaches a method for augmented reality guided instrument positioning, 
comprising the steps of: rendering at least one graphics path guide (see figure 6) 
for indicating a path for an instrument to follow to a target (see section 3 which 
describes determining the path the needle will follow based upon 
the target and a given anatomical landmark)] identifying at least one axis 
marker on the instrument that marks an axis of the instrument (the robot arm and 
needle have an inherent axis), the axis disposed from a front portion to a back 
portion of the instrument, the front portion corresponding to the point on the target; and 
aligning the at least one axis marker with respect to the at least one graphics guide to 
align the instrument to the path (once the system has determine the path 
the needle will take and displayed the path graphically, see 
figure 6, then the system will align the needle to the path or 
allow the surgeon to align the needle to the determined point on 
the patient and the path). 

Claim 16: 

Bascle teaches the method according to claim 13, further comprising the step of 
selecting an existing feature of the instrument to be the at least one axis marker (the 
existing feature is the passive or active robot arm). 
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Claim 17: 

Bascle teaches the method according to claim 13, further comprising the step of 
designing the instrument to include the at least one axis marker (robots are 
designed to have the arms defined by an axis, the term marker 
can be a physical thing or a virtual thing). 

Claim 18: 

Bascle teaches the method according to claim 13, further comprising the step of adding 
the at least one axis marker to a structure of the instrument (the needle is added 
to the robot arm which has the inherent axis). 

Claim 19: 

Bascle teaches the method according to claim 13, wherein the at least one axis marker 
is an elongated member (the robot arm is an elongated member). 

Claim 20: 

Bascle teaches the method according to claim 13, wherein the at least one axis marker 
has a circular shape, and is centered on the axis of the instrument (the needle 
itself is an axis marker and since a needle is hollow it is a 
circular marker centered on the axis of the needle). 
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Claim 21: 

Bascle teaches the method according to claim 13, wherein the at least one axis marker 
is a cross comprised of an intersection of at least two lines (the term marker is a 
broad term and is met by the geometric marker robots use to 
control the move of their arms, robots use rotation about an 
origin having two perpendicular lines, Cartesian coordinates, to 
determine the location of the arm and to position the arm), the 
intersection being centered on the axis of the instrument (needle, if the needle 
is not centered then the position of the needle will have to be 
offset relative to the arm, thus, centering the arm onto the 
needle). 

Claim 22: 

Bascle teaches the method according to claim 13, wherein the at least one axis marker 
comprises at least two axis markers for controlling alignment of the instrument along a 
line of sight (an axis marker is a broad term and two axis markers for 
controlling alignment of the instrument is still broad, a first 
interpretation of what is meant by the claimed axis marker is one 
axis marker is the robot arm and the other is the needle another 
interpretation is the sensors in the robot arm for determining 
the axis are the claimed two axis markers since they do mark an 
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axis of the instrument). 



Copending Applications 



8. A review of US Patent Application Publications reveals at least two of applicants 
copending applications, 09/818,388 and 09/818,122, having claims which are very 
close to the claims in this application. The examiner has reviewed the corresponding 
Patent Application Publications to review the claims, one of which has had no 
amendments made to the application claims. Applicant is requested to maintain a clear 
line of demarcation between applicant's copending applications and this application to 
avoid double patenting issues. 



9. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Andrei State et al., Technologies for Augmented Reality Systems: Realizing 
Ultrasound-Guided Needle Biopsies, 1996, ACM Press, Proceedings of the 23 rd annual 
conference on Computer Graphics and Interactive Techniques, pages 439-446, teaches 
a way to obtain proper occlusion relationships between real and virtual objects in 
augmented reality. 

Matthias M. Wloka et al., Resolving Occlusion in Augmented Reality, ACM Press, 
Proceedings of the 1995 Symposium on Interactive 3D Graphics, pages 5-12, teaches 



Prior Art 
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resolving occlusion in augmented reality by using an algorithm that assigns depth 
values to each pixel in a pair of stereo video images in near-real-time. 

S. Otmane et al., Active Virtual Guides as an Apparatus for Augmented Reality 
Based Telemanipulation System on the Internet, Simulation Symposium, 2000, April 16- 
20, 2000, pages 185-191 , teaches using line segments, a plan or any arbitrary volume 
as a virtual guide. 

Frank Sauer, et al., Augmented Workspace: designing an AR testbed, 
Proceedings and ACM International Symposium on Augmented Reality, 2000, October 
5-6, 2000, pages 47-53, teaches using a virtual camera to render the video background 
and placing virtual objects onto the background and in the last paragraph in section 10 
describes how the virtual objects occludes the background even when the virtual object 
is behind the background and states these issues have to be addressed on the way 
towards real applications. 

Sundareswaran et al., U.S. Patent No. 6,330,356, teaches an augmented reality 
system. 

Keller et al., U.S. Patent No. 6,503,195, teaches with regard to figures 8-1 0b an 
augmented reality display system where a video of the need outside the patient's body 
is displayed and a virtual needle showing where the needle is in the patient's body. 
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Allowable Subject Matter 

10. Claims 8, 9, 14, 15, 28-34, and 36-39 are objected to as being dependent upon a 
rejected base claim, but would be allowable if rewritten in independent form including all 
of the limitations of the base claim and any intervening claims. 

The following is a statement of reasons for the indication of allowable subject 
matter: 
Claims 8-9: 

The prior art of record fails to reach or suggest the plurality of graphical markers 
comprises at least one ring centered on the path. 
Claims 28-30: 

The prior art of record fails to reach or suggest the virtual instrument's 3D structure 
comprises a plurality of markers centered on and distributed along an axis of the virtual 
instrument. 
Claims 31-34: 

The prior art of record fails to reach or suggest choosing an orientation of the graphical 
representation around the virtual line of sight according to a pose of a user with respect 
to the actual target. 
Claims 36-39: 

The prior art of record fails to reach or suggest the graphical information about the 
distance is overlaid onto the graphical representation. 
Claims 14 and 15: 
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The prior art of record fails to reach or suggest the step of constructing and rendering 
the at least one axis marker as graphical marker. 

1 1 . Any inquiry concerning this communication or earlier communications from 
the examiner should be directed to Jeffery A. Brier whose telephone number is (703) 
305-4723. The examiner can normally be reached on M-F from 6:30 to 3:00. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Michael Razavi, can be reached at (703) 305-4713). 

Any response to this action should be mailed to: 

Commissioner of Patents and Trademarks 
Washington, D.C. 20231 

or faxed to: 



Hand-delivered responses should be brought to Crystal Park II, 2121 Crystal 
Drive, Arlington, VA, Sixth Floor (Receptionist). 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the Technology Center 2600 Customer Service Office 
whose telephone number is (703) 306-0377. 



(703) 872-9306 (for Technology Center 2600 only) 




Jeffery A Brier 
Primary Examiner 
Art Unit 2672 



